test(0121): k6 script and load-test plan for the 100 req/s AC - #234

MEDIUM — the authoritative run command (line 18) still pins -e ASSET=native, which contradicts the regime table this PR adds.

Line 88 below now says -e ASSET=native is the cache regime, explicitly not the AC scenario. But the headline command at line 18 — the one anyone will copy for the milestone run — still carries the flag.

Failure: someone runs the documented authoritative command, measures the API Gateway cache at ~0.1 % miss, and reports the p95 as the 100 req/s AC result. That is precisely the "configuration artefact as an SLO result" failure the prose here warns about.

Dropping -e ASSET=native from line 18 makes the headline command use the new 20-asset default.

LOW — the "1000+ assets → 100 % miss" row is wrong at exactly 1000.

Asset selection is deterministic round-robin (iterationInTest % pool.length), so at RATE=100 a 1000-asset pool returns to each asset once every exactly 10.0 s — which is the cache TTL (CACHE_TTL.price). Hit vs miss becomes a timing coin-flip, so the "real data path" regime silently measures a mix and the p95 is neither number.

The real condition is pool ≫ RATE × TTL. Worth stating it that way, or naming a value with margin (≥ 2000 at 100 req/s), rather than a round number that sits exactly on the boundary.

LOW — open() copies the pool into every VU; SharedArray would not.

Across both scenarios maxVUs totals 400, and k6 duplicates open() data per VU — that is the documented reason SharedArray from k6/data exists.

Irrelevant for the 3 KB default file, but the README recommends a 1000+ asset pool for the real-data-path regime, where this multiplies into real generator-side memory. The failure is indirect and misleading: a generator that becomes the bottleneck produces dropped iterations, the dropped_iterations threshold correctly fails, and the operator is sent to bump MAX_VUS — making it worse — rather than to the cause.

LOW/MEDIUM — the warmup does not warm the concurrency the main phase needs.

rate: RATE/10 is 10 req/s at the default, which sustains roughly one concurrent Lambda execution. The main phase at 100 req/s needs ~9-10. So around nine containers still cold-start inside the phase:main window — exactly what the header comment (lines 34-39) says the warmup exists to prevent.

Warming at the full RATE for a shorter window would actually pre-scale the concurrency.

In fairness this also undercuts the comment's own premise: ~9 cold starts in 30 000 samples will not move p95, so the effect being guarded against is smaller than described either way.

LOW — -e WARMUP=0s does not skip the warmup, it prevents the run from starting.

WARMUP is documented as a knob, and 0s is the obvious way to skip it for a quick local iteration. But it gives this scenario duration: '0s', which k6 rejects at config validation — so the test does not start at all, with an error that points at the scenario rather than at the flag.

Either guard the scenario behind a truthiness check on WARMUP, or document that it must be > 0.

MEDIUM — there is no checks threshold, so the new body validation cannot fail the run.

k6 check results do not affect the exit code; only thresholds do. http_req_failed{phase:main} covers status via expectedStatuses(200), but nothing here covers the body.

Failure: a regression serves 200 {"price_usd": null} (or a numeric price_usd instead of the JSON string) for all 30 000 requests. Every threshold passes, k6 exits 0, and the harness certifies the SLO as met against a broken endpoint.

The comment at line 149 — "A body-read failure must not pass as a slow 200" — asserts a guarantee the code does not currently provide. checks: ['rate>0.99'] provides it.

LOW — dropped_iterations is unscoped, so a warmup drop fails the run.

The warmup scenario is deliberately excluded from every other threshold, but count<1 here is global and covers phase:warmup too. A drop during warmup — when containers are still scaling, which is the entire point of that phase — fails the run and is reported as "did NOT sustain 100 req/s".

'dropped_iterations{phase:main}': ['count<1'] matches the comment's own description of the measured window.

MEDIUM — setup() will time out in the wide-pool regime the README recommends.

k6's default setupTimeout is 60 s and this probe loop is strictly sequential. The README (line 88) documents 1000+ assets as the way to measure the real data path; 1000 sequential probes at the measured ~83 ms p95 is roughly 83 s, so setup() aborts the entire run before a single measured request is sent.

Set setupTimeout in options, or probe concurrently / sample the pool — before anyone tries the regime the README points them at.

LOW/MEDIUM — any single non-200 is treated as permanent unservability, and the abort message misdirects.

A transient 429 or 5xx during the probe silently shrinks the pool, and the drop is reported as though the asset were dead.

The sharpest case is one this README already warns about: pricing-api-free-production is capped at 1 req/s (lines 24-30). Run with a free-tier key and this sequential probe (~12 req/s) is throttled on nearly every asset — the pool collapses and the script aborts with pool: no asset answered 200 — nothing to measure, pointing the operator at the data rather than at the key they used.

Distinguish 404 (drop) from 429/5xx (retry, or fail loudly), and name the status in the abort message.

MEDIUM — this X-Request-Id → ClickHouse log_comment join does not exist.

grep -rniE "log_comment|x-request-id" over the repo (.rs, .ts, .md) returns nothing: the API never reads this header and never sets log_comment. The comment hedges with "when the API runs with request-id logging", but no such mode exists anywhere in the codebase, so the header is inert.

That would be harmless as a forward-looking stub, except the PR body and the task-file history both record it as a delivered capability ("X-Request-Id per request for a ClickHouse log_comment join") — which will send the next person hunting for a join that cannot be made.

Either plumb the header through the handler into log_comment, or drop the claim from the task file. Note also that with the 20-asset pool ~98 % of main-phase requests are served from the gateway cache and never reach the Lambda, so even once plumbed the join would only cover the ~2 % that miss.